Resilient Binary Neural Network

Binary neural networks (BNNs) have received ever-increasing popularity for their great capability of reducing storage burden as well quickening inference time. However, there is a severe performance drop compared with {real-valued} networks, due to its intrinsic frequent weight oscillation during training. In this paper, we introduce Resilient Neural Network (ReBNN) mitigate the better BNNs' We identify that mainly stems from non-parametric scaling factor. To address issue, propose parameterize factor and weighted reconstruction loss build an adaptive training objective. For first time, show controlled by balanced parameter attached loss, which provides theoretical foundation it in back propagation. Based on this, learn our ReBNN calculating based maximum magnitude, can effectively resilient process. Extensive experiments are conducted upon various network models, such ResNet Faster-RCNN computer vision, BERT natural language processing. The results demonstrate overwhelming over prior arts. example, achieves 66.9% Top-1 accuracy ResNet-18 backbone ImageNet dataset, surpassing existing state-of-the-arts significant margin. Our code open-sourced at https://github.com/SteveTsui/ReBNN.